Asymptotic and non-asymptotic convergence properties of stochastic approximation with controlled Markov noise without ensuring stability

This paper talks about both the asymptotic and non-asymptotic convergence properties of stochastic approximation algorithms with controlled Markov noise when stability of the iterates (which is an important condition for almost sure convergence) is hard to prove. We achieve the same by giving a lower bound on the lock-in probability of such frameworks i.e. the probability of convergence to a specific attractor of the o.d.e. limit given that the iterates visit its domain of attraction after a sufficiently large number of iterations n0. With the more general assumption of controlled Markov noise supported on a bounded subset of the Euclidean space, we recover the same bound available in literature for the case of controlled i.i.d noise (i.e. martingale difference noise). We use these results to prove almost sure convergence of the iterates to the specified attractor when sufficient conditions to guarantee asymptotic tightness of the iterates are satisfied. Another important corollary of our results is a statement similar to the well-known Kushner-Clarke Lemma for stochastic approximation algorithm with controlled Markov noise, however without the assumption of the stability of the iterates. This is, for step-size sequence { 1 nk }, 1 2 < k ≤ 1 and 1 n(logn)k , k ≤ 1, the almost sure convergence to a local attractor provided that the iterates belong to some open set infinitely often such that the open set contains the local attractor and its compact closure is in the domain of attraction of the local attractor. Additionally, we show that such results can be used to derive a sample complexity estimate of such recursions which is then used in predicting the optimal step-size.